Characterizing Model Performance in the Feature Space

A fundamental problem in machine learning is understanding the conditions for which a learning algorithm works well. Understanding an algorithm's strengths and weaknesses and being able to compare two algorithms with each other are necessary for designers to develop (or select) learning algorithms for a speci c problem. Generally, one can attempt to analyze and understand algorithms either theoretically or empirically. Theoretical analyses of machine learning algorithms have usually resulted in weak performance guarantees that are not much use to a practitioner. Algorithms are typically proven to be asymptotically consistent (i.e., will achieve the Bayes optimal error rate given enough training examples) or that the algorithm can be used to PAC (Probably Approximately Correct) learn a given concept (Valiant, 1984). Another approach is to analyze average case behavior under speci c distributional assumptions, such as learning m-of-n concepts (Langley & Sage, 1999). Although these analyses are useful in understanding the general behavior of an algorithm, they are unable to provide guidance to the designer in the form of speci c predictions of an algorithm's performance with a given problem. Thus most researchers and practitioners resort to empirical evaluation to understand the interaction between learning algorithms and a domain. Unfortunately, most evaluation methods give very little information to the designer. For example, the most common method of empirically evaluating a classi er is to examine its error, or more generally loss, and many comparisons of algorithms use only this metric. Loss can easily be estimated by using a test set or cross-validation. However because loss is a single number, it reveals little about the algorithm except gross performance on the domain. Other researchers have tried to learn when a classi cation algorithm is appropriate for a problem domain based on characteristics of the data set. The basic idea is to use properties such as the number of features, number of classes, or the number of instances to learn by inspection or through automated analysis when an algorithm is appropriate. For example, Aha (1992) used a rule learner to automatically derive results such as the following.